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Abstract 

A  goal  of  this  paper  is  to  explore  differ¬ 
ent  ways  of  implementing  distributed  simu¬ 
lations.  Distributed  simulation  grew  out  of 
sequential  simulation,  and  it  is  possible  that 
the  way  we  think  about  distributed  simulation 
is  unduly  influenced  by  its  sequential  origins. 
To  free  ourselves  from  unnecessary  restrictions 
on  the  way  we  design  distributed  simulations, 
in  this  paper  we  define  the  distributed  simu¬ 
lation  problem  somewhat  differently  than  in 
the  literature.  We  propose  the  concepts  of 
“knowledge”  and  “conditional  knowledge”,  to 
help  us  obtain  a  general  framework  to  rea¬ 
son  about  distributed  simulations  without  too 
close  a  coupling  with  any  specific  implementa¬ 
tion  method.  The  framework  appears  helpful 
in  designing  new  ways  of  distributed  simula¬ 
tions. 

*  supported  by  ONR  Grant  Nos.  N 00014- 86- K- 
0466,  N00014-87-K-0510,  and  the  Sherman  Fairchild 
Foundation 


Empirical  studies  of  distributed  simulations 
report  widely  varying  results:  some  studies  re¬ 
port  improvements  in  speed  that  are  almost 
linearly  proportional  to  the  number  of  com¬ 
puters  in  the  system,  while  other  studies  re¬ 
port  that  distributed  simulation  is  even  slower 
than  sequential  simulation.  The  framework 
proposed  in  this  paper  seems  to  help  in  ex¬ 
plaining  the  wide  differences  observed  in  em 
pirical  studies. 


Using  our  framework,  we  attempt  to  suggest 
properties  that  efficient  “general-purpose”  dis¬ 
tributed  discrete-event  simulations  must  have. 


The  paper  assumes  little  prior  knowledge  of 
the  literature  on  simulation  or  distributed  sys¬ 
tems.  We  hope  that  the  paper  will  serve  as  a 
tutorial  in  addition  to  providing  additional  in¬ 
sight. 


1  INTRODUCTION 

This  paper  is  an  informal  exploration  of 
methods  for  implementing  distributed  simula¬ 
tors  as  described  in  [Chandy  and  Misra  1979, 
Chandy  and  Misra  1981,  and  Jefferson  1985]. 
We  attempt  to  explore  concepts  that  appear 
helpful  in  thinking  about  the  problem.  Our 
intent  is  to  understand  the  concepts  at  an  in¬ 
tuitive  level  rather  than  to  propose  a  mathe¬ 
matical  theory  or  to  describe  an  implementa¬ 
tion.  In  this  sense,  the  paper  describes  “work 
in  progress”  or  “ideas  in  gestation” .  Section  2 
describes  “conditional  events”  -  the  basis  for 
sequential  simulation.  Then  “unconditional 
events”  -  a  basis  for  distributed  simulation  is 
discussed.  Two  examples  are  used  to  illustrate 
these  concepts:  the  first  is  an  ultra-simplistic 
model  of  a  battlefield,  and  the  second  is  a 
model  of  a  job  shop  or  assembly  line  [Misra 
1986].  The  battle  field  example  is  used  to  sug¬ 
gest  that  the  unconditional-event  paradigm  of 
distributed  simulation  may  have  limitations. 

The  concept  of  “knowledge”  in  distributed 
simulation  is  introduced  in  Section  3.  The  def¬ 
inition  employed  here  is  different  from  the  one 
employed  in  the  literature  on  distributed  sys¬ 
tems  [Chandy  and  Misra  1986,  Halpern  and 
Moses  1984].  The  concept  is  extended  to  “con¬ 
ditional  knowledge”  in  Section  4.  The  con¬ 
cepts  of  conditional  and  unconditional  knowl¬ 
edge  form  the  framework  which  is  employed  to 
propose  methods  of  implementing  simulations 
on  parallel  machines  in  Section  5.  Section  6 
is  a  conclusion  in  which  we  use  the  framework 
to  propose  desirable  properties  of  distributed 
simulators. 

2  CONDITIONAL  EVENTS  AND  SE¬ 
QUENTIAL  SIMULATION 


Consider  a  sequential  simulation  of  a  net¬ 
work  of  processes.  The  computation  to  be 
simulated  consists  of  a  sequence  of  “events”, 
where  an  event  is  a  change  in  (global)  system 
state.  Each  event  is  “initiated”  by  one  pro¬ 
cess,  in  the  sense  that  the  precondition  for  the 
event  is  determined  by  the  state  of  precisely 
one  process.  However,  the  occurrence  of  the 
event  may  change  the  states  of  an  arbitrary 
number  of  processes. 

A  battle  between  two  submarines,  for  ex¬ 
ample,  can  be  thought  of  as  a  network  of  two 
processes  where  each  “sub”  is  a  process.  Nei¬ 
ther  sub  knows  where  the  other  one  is,  but  it 
may  guess  the  other’s  location  from  informa¬ 
tion  such  as  sonar  readings.  Firing  a  missile, 
or  changing  velocity,  are  examples  of  events. 
Whether  a  missile  is  fired  by  a  sub,  depends 
only  on  the  state  of  the  sub,  and  does  not  de¬ 
pend  on  the  state  of  the  total  system.  In  par¬ 
ticular,  whether  a  sub  fires  a  missile,  depends 
on  the  information  it  has  obtained  from  its 
sensors  about  the  location  of  the  other  sub, 
but  it  does  not  depend  on  the  actual  location 
of  the  other  sub.  The  occurrence  of  an  event 
may  change  the  state  of  the  entire  system. 
Thus,  the  simulation  model  allows  the  firing 
of  a  missile  by  one  sub  to  destroy  the  other 
sub  instantaneously.  (It  can  be  argued  that 
the  simulation  model  is  too  powerful  because 
missiles  take  non-zero  time  to  travel.  A  more 
general  argument  is  that  events  cause  only  “lo¬ 
cal”  as  opposed  to  global  state  changes.  How¬ 
ever,  an  event  that  is  initiated  at  a  single  pro¬ 
cess,  but  which  may  change  the  states  of  all 
processes,  is  a  useful  concept  in  building  mod¬ 
els  that  are,  after  all,  only  approximations  of 
reality  In  any  case  it  is  not  necessary  to  use  all 
the  power  of  the  simulator.) 

The  central  data  structures  in  a  sequential 
simulation  are  the  event  list  and  a  variable, 
“clock” ,  which  is  a  non-negative  integer  repre- 


senting  time.  The  event  list  contains  one  entry 
for  each  process.  An  entry  e  is  a  tuple  (e.name, 
e.T)  where  e.name  identifies  the  type  of  the 
event,  and  e.T  is  a  non-negative  integer  repre¬ 
senting  time,  where  e.T  >  clock.  The  meaning 
of  an  entry  (e.name,  e.T)  corresponding  to  a 
process  p  is  as  follows: 

For  all  t'  where  clock  <=  t'  <  e.T  : 

IF  all  other  processes  do  not  initiate  events 
in  the  interval  (clock,  t') 

THEN  no  event  is  initiated  by  p  in  the  in¬ 
terval  (clock,  t'  + 1) 
and 

IF  all  other  processes  do  not  initiate  events 
in  the  interval  (clock,  e.T) 

THEN  p  initiates  event  e.name  at  time  e.T. 

Note:  the  interval  (clock,  <')  is  the  open  in¬ 
terval,  i.e.,  it  is  the  interval  after  clock  and 
before  if.  The  action  taken  by  a  process  at 
time  t  may  depend  on  actions  of  processes  at 
times  earlier  than  t,  but  it  does  not  depend 
on  actions  of  other  processes  at  time  t  (where 
an  action  is  either  initiating  a  specific  event  or 
not  initiating  any  event).  It  follows  that  the 
time  at  which  the  next  event  is  initiated  is  the 
smallest  value  of  e.T  over  all  events  e  in  the 
event  list. 

Sequential  simulation  is  a  repetition  of  the 
following  sequence  of  actions: 

•  Determine  the  entries  e  in  the  event  list 
with  the  smallest  e.T; 

•  execute  these  events,  i.e.,  change  the  sys¬ 
tem  state  appropriately; 

•  recompute  the  entry  in  the  event  list  cor¬ 
responding  to  each  process. 

In  a  sequential  simulation  of  the  submar 
rine  battle,  the  event  list  consists  of  an  entry 
(e.name,  e.T)  for  each  sub,  where  e.T  is  the 
time  at  which  the  sub  next  initiates  an  event 


provided  the  other  sub  does  not  initiate  an 
earlier  event.  The  simulation  proceeds  by  exe¬ 
cuting  the  event  corresponding  to  the  smaller 
of  the  e.T  values,  and  executing  an  event  may 
change  the  states  of  both  subs.  The  new  entry, 
in  the  event  list,  for  each  sub  is  then  recom¬ 
puted.  The  cycle  is  then  repeated. 

Consider  a  direct  implementation  of  the  se¬ 
quential  simulation  of  the  submarine  battle  on 
a  parallel  machine  with  two  processors,  with 
one  processor  for  each  sub.  The  first  step  is 
to  determine  which  of  the  two  processors  has 
the  smaller  value  of  e.T.  The  processor  with 
the  smaller  e.T  communicates  the  correspond¬ 
ing  e.name  to  the  other  processor.  Both  pro¬ 
cessors  determine  their  next  states  in  parallel, 
and  both  processors  recompute  their  new  en¬ 
tries  in  the  event  list  in  parallel. 

The  structure  of  the  computation  is  as  fol¬ 
lows: 

•  determine  the  minimum  over  all  entries  e 
in  the  event  list  of  e.T, 

•  broadcast  the  name  of  the  next  event  to 
all  processors, 

•  and  each  processor  determines  its  new 
state  and  event-list  entry  in  parallel. 

In  general,  if  an  event  changes  the  states 
of  most  of  the  processes,  then  parallelism  can 
be  exploited  in  the  computations  of  the  next 
states  of  the  processes.  On  the  other  hand,  if 
an  event  changes  the  states  of  a  very  small 
fraction  of  the  processes,  then  there  is  less 
scope  for  exploiting  parallelism  with  this  ap¬ 
proach.  Let  us  next  consider  the  simulation 
of  quite  a  different  system  to  get  a  better  un¬ 
derstanding  of  the  limits  of  directly  mapping 
sequential  simulation  programs  on  parallel  ar¬ 
chitectures. 

A  simple  job-shop  consists  of  a  source  of  jobs 
followed  by  a  sequence  of  queues,  represent- 


mg  workstations,  followed  by  a  job  sink.  A 
diagrammatic  representation,  where  W  repre¬ 
sents  a  workstation  is  given  next. 

source  —*■  W  — ►  W  — ►  W  —*■  sink 

Assume  that  each  workstation  processes  the 
jobs  it  receives  in  the  order  received.  In  the 
simulation  there  are  processes  corresponding 
to  the  source,  the  sink  and  to  each  of  the 
workstations.  An  event  is  a  transmission  of 
a  job,  from  the  source  to  a  workstation,  be¬ 
tween  workstations,  or  from  a  workstation  to 
the  sink.  Obviously  an  event  in  the  job-shop 
changes  the  state  of  precisely  two  processes.  A 
direct  implementation  of  the  sequential  simu¬ 
lation  of  a  job  shop  on  a  parallel  architecture 
is  not  likely  to  result  in  substantial  speed-up, 
because  there  are  at  most  two  processors  exe¬ 
cuting  at  any  time.  Even  if  we  were  simulating 
a  job  shop  with  a  large  number  of  workstations 
on  a  parallel  architecture  with  one  processor 
for  each  workstation,  we  would  not  expect  to 
do  better  than  double  the  speed  of  a  sequential 
simulation. 

The  moral  of  this  little  discussion  is  the  ob¬ 
vious  one — whether  a  given  method  of  dis¬ 
tributed  simulation  works  efficiently  depends 
on  the  sytem  being  simulated. 

Let  us  generalize  the  event  list  to  have  two 
kinds  of  entries:  unconditional  entries  and 
conditional  entries.  The  entries  e,  in  the  event 
list,  described  earlier,  are  conditional  entries. 
An  unconditional  entry  u,  in  the  event  list,  is 
a  tuple  (u.name,  u.T)  where  u.name  identifies 
the  event  and  u.T  is  a  time  and  u.T  >  clock. 
The  meaning  of  the  entry  is  as  follows:  an  un¬ 
conditional  entry  (u.name,  u.T)  for  a  process  p 
means  that  process  p  will  not  initiate  an  event 
in  the  interval  (clock,  u.T)  and  process  p  ini¬ 
tiates  event  u.name  at  time  u.T;  furthermore, 
the  actions  taken  by  process  p  in  the  interval 
(clock,  u.T)  are  INDEPENDENT  of  the  ac¬ 


tions  taken  by  other  processes  in  the  interval. 
The  critical  difference  between  unconditional 
and  conditonal  entries  is  that  in  the  case  of 
a  conditional  entry  (e.name,  e.T)  for  a  pro¬ 
cess  p,  the  actions  taken  by  p  in  the  inter¬ 
val  (clock,  e.T)  MAY  DEPEND  on  the  actions 
of  other  processes  in  the  interval  (clock,  e.T). 
Of  course,  every  unconditional  entry  may  be 
treated  as  a  conditional  entry,  but  conditional 
entries  cannot  (in  general)  be  treated  as  un¬ 
conditional  entries. 

We  introduce  a  special  event  called  the  null 
event,  which  does  not  change  the  state  of  the 
system.  Thus  if  there  is  an  unconditional  en¬ 
try  (u.name,  u.T)  for  a  process  p,  in  the  event 
list,  where  u.name  is  the  null  event,  then  it 
means  that  process  p  does  not  execute  any 
“real”  event  in  the  interval  (clock,  u.T],  where 
(clock,  u.T]  is  the  interval  that  excludes  clock, 
but  includes  u.T. 

Consider  the  source  in  the  job-shop  exam¬ 
ple.  The  events  it  initiates  correspond  to  the 
jobs  that  arrive  at  the  shop.  Presuming  that 
job  arrivals  do  not  depend  on  the  manner  in 
which  jobs  are  processed  in  the  shop,  the  ac¬ 
tions  taken  by  the  source  do  not  depend  on 
the  actions  of  other  processes.  Therefore,  all 
the  entries  in  the  event  list  corresponding  to 
the  source  are  unconditioned  events.  Further¬ 
more,  the  source  may  produce  a  stream  of  un¬ 
conditioned  entries  corresponding  to  the  times 
at  which  a  stream  of  jobs  arrives  at  the  shop. 
There  is  no  reason  for  the  execution  of  the  sim¬ 
ulator  of  the  source  to  depend  on  the  execution 
of  the  simulator  of  any  other  process  (provided 
that  there  is  enough  memory  to  store  the  un¬ 
conditional  entries). 

Now  consider  a  workstation  in  the  job-shop. 
Assume  that  with  each  job  is  a  work-order  that 
specifies  the  amount  of  processing  time  that 
the  job  requires  at  the  workstation.  If  a  work¬ 
station  is  processing  a  job  at  some  time,  then 


we  know  the  time  at  which  the  workstation 
will  complete  processing  the  job,  and  this  time 
is  independent  of  the  actions  taken  by  other 
processes.  Therefore,  all  entries  representing 
the  movement  of  jobs  sure  unconditional. 

Are  there  any  conditional  entries  in  a  se¬ 
quential  simulation  of  a  job-shop?  Yes:  A 
process  simulating  an  idle  workstation  posts 
a  conditional  entry  (null,  infinity)  to  indicate 
that  IF  the  workstation  does  not  receive  any 
more  jobs  THEN  the  workstation  will  not  out¬ 
put  any  more  jobs. 

One  way  of  simulating  a  job-shop  on  a  par¬ 
allel  machine  is  as  follows.  The  event  list  con¬ 
tains  a  stream  (a  queue)  of  unconditional  en¬ 
tries  from  each  process.  Each  queue  of  entries 
in  the  event  list  corresponds  to  a  queue  of  jobs 
for  a  workstation.  The  processor  simulating 
a  workstation  removes  the  entry  at  the  head 
of  its  input  queue,  and  processes  it,  i.e.,  adds 
an  entry  to  its  output  queue.  For  example, 
if  the  entry  at  the  head  of  the  input  queue  is 
(u.name,  t)  where  u.name  is  a  work-order  that 
specifies  that  the  job  gets  f  units  of  processing 
time  at  the  workstation,  and  if  the  workstation 
is  idle  at  time  t,  then  the  processor  simulating 
the  workstation  puts  (u'.name,  t  + t')  in  its 
output  queue,  where  u'.name  describes  the  re¬ 
maining  work-order  for  that  job.  No  synchro¬ 
nization  is  necessary.  If  at  time  t,  the  work¬ 
station  has  t"  units  of  processing  to  complete 
before  it  can  get  to  the  newly  arrived  job,  then 
the  processor  outputs  (u'.name,<  +  t'  + 1"). 

This  method  is  faster  than  executing  the 
simulation  on  a  sequential  processor.  Indeed, 
this  simulation  is  ideal  for  a  pipeline  architec¬ 
ture  with  one  stage  in  the  pipeline  simulating 
one  process  in  the  job-shop.  Once  the  pipe  is 
full,  a  great  deal  of  concurrency  is  exploited. 

Now  let  us  attempt  to  use  unconditional  en¬ 
tries,  exclusively,  for  the  battlefield  problem. 


How  fax  can  we  predict  the  actions  taken  by 
one  sub,  that  are  independent  of  the  actions 
taken  by  the  other  sub?  One  presumes  that 
in  a  battlefield,  one  sub  monitors  the  other 
continuously,  and  therefore  the  only  prediction 
that  we  can  make  about  one  sub,  given  ONLY 
the  state  of  that  sub,  is  what  actions  it  will 
take  in  the  very  next  instant.  In  other  words, 
an  unconditional  entry  is  limited  to  the  form 
(u.name,  u.T)  where  u.T  =  (clock+1).  To  pre¬ 
dict  the  behavior  of  one  sub  over  a  longer  hori¬ 
zon,  we  need  to  have  information  about  both 
subs. 

The  unconditional-entry  approach,  applied 
to  the  battlefield,  results  in  the  following  type 
of  computation:  The  event  list  consists  of  two 
entries,  one  unconditional  entry  per  sub.  Each 
of  the  entries  is  of  the  form  (u.name,  u.T) 
where  u.T  =  (clock  +1).  Each  process  com¬ 
putes  its  state  at  time  clock+1,  then  the  clock 
is  incremented  by  1,  and  each  process  deter¬ 
mines  its  new  unconditional  entry  (u.name, 
u.T),  where  unfortunately,  u.T  =  (clock+1). 
The  resulting  computation  is  a  time-driven 
simulation:  each  step  of  the  computation  cor¬ 
responds  to  incrementing  the  clock  by  1.  Sim¬ 
ulating  the  battlefield,  using  unconditional  en¬ 
tries  alone,  on  a  parallel  processor  may  result 
in  slower  execution  than  discrete-event  simu¬ 
lation  on  a  sequential  processor. 

What  have  we  learned  from  these  exam¬ 
ples?  The  approach  of  using  conditional  en¬ 
tries  alone,  appears  appropriate  for  parallel 
discrete-event  simulation  of  our  example  of  the 
battlefield.  The  approach  of  using  uncondi¬ 
tional  entries  alone,  appears  appropriate  for 
parallel  simulation  of  our  example  of  a  job- 
shop.  Time-driven  simulation  is  inherently 
parallel,  and  indeed  it  may  be  the  simula- 
tion  method  of  choice  because,  in  many  sys¬ 
tems,  the  state  changes  continuously  over  time 
rather  than  at  discrete  instants. 


It  seems  apparent  that  a  simulation  method 
that  combines  conditional  and  unconditional 
entries  is  worth  studying  because  it  is  likely 
to  be  efficient  for  a  wider  class  of  problems. 
Before  exploring  this  possibility  let  us  define 
“knowledge  in  distributed  simulations”,  and 
use  the  concept  to  gain  insight  into  the  prob¬ 
lem  of  distributed  simulation. 

3  KNOWLEDGE  AND  SIMULATION 

Our  goal,  in  this  section,  is  to  gain  an  un¬ 
derstanding  of  what  makes  for  efficient  dis¬ 
tributed  simulations.  However,  to  gain  this 
understanding,  we  shall  define  some  terms  for¬ 
mally.  For  brevity,  the  system  that  is  being 
simulated  is  referred  to  as  “the  system”.  The 
processes  in  the  system  are  called  physical  pro¬ 
cesses  (or  PPs)  to  distinguish  them  from  pro¬ 
cesses  in  the  simulator.  A  system  is  a  set  of 
PPs  and  a  set  E  of  events.  Set  E  includes 
the  null  event.  A  PP  is  a  set  S  of  process 
states,  an  initial  state  in  that  set,  an  initial 
event  (which  is  an  element  of  E),  a  next-  state 
function  /,  and  a  next-event  function  g,  where: 

f  ::  S  x  D  —*  S, 

where  D  is  the  powerset  of  E, 
g  ::  S  x  D  — ►  E. 

The  meaning  of  /  is  as  follows.  If  the  PP  is 
in  state  s  at  some  time  t,  and  the  set  of  events 
d  occurs  at  time  t,  then  the  state  of  the  PP, 
at  time  t  + 1  is  /(«,  d).  The  meaning  of  g  is  as 
follows.  If  the  PP  is  in  state  s  at  time  t,  and 
the  set  of  events  d  occurs  at  time  t,  then  the 
PP  initiates  event  g(s,  d)  at  time  t  +  1.  Note 
that  g(s,d)  may  be  the  null  event.  A  PP  is  in 
its  initial  state  and  initiates  its  initial  event  at 
time  0. 

The  PPs  we  have  defined  are  determinis¬ 
tic:  their  future  behaviors  are  functions  of 


their  histories.  In  the  real  world,  PPs  may 
be  nondeterministic;  for  example  a  PP  may 
toss  a  coin  to  determine  whether  to  initiate 
one  event  or  another.  In  a  simulator,  ran¬ 
domness  is  modeled  by  employing  a  pseudo¬ 
random  number  generator.  Given  the  value 
of  the  seed  of  a  pseudo-random  number  gen¬ 
erator,  the  sequence  of  numbers  generated  is 
deterministic.  Given  the  values  of  the  seeds, 
each  execution  of  a  simulator  is  deterministic. 
In  analogy,  we  define  a  PP  in  terms  of  a  sin¬ 
gle  behavior,  though  in  reality  it  may  exhibit 
many  behaviors. 

For  example,  a  partial  specification  of  a  sub¬ 
marine  in  a  simplistic  video  game  is  as  follows. 

5  =  {on,  off} 

E  =  {null0,fire0,nulll,firel} 

where  nullO,  fireO  means  that  subO  initiates  the 
null  event,  fires  a  missile  respectively;  nulll, 
firel  are  analogous  events  for  subl. 

/(off,d)  =  off  for  all  d 

/(on,  {nullO, nulll})  =  on 
/(on,  {nullO,  firel})  =  off 
/(on,  {fireO,  nulll})  =  on 
/(on,  {fireO,  firel})  =  on 

Thus  the  submarine  we  are  defining  has  the 
property  that  once  it  is  “off”  it  remains  “off” , 
and  if  it  is  “on”  it  remains  “on”  except  for 
the  case  that  it  does  not  fire  and  the  other  sub 
does.  (Presumably,  if  both  subs  fire  simultane¬ 
ously,  the  missiles  destroy  each  other  without 
damaging  the  subs,  which  seems  to  occur  quite 
often  in  video  games.) 

A  system  state  is  a  tuple;  its  components 
are  process  states.  The  behavior  of  the  system 


is  an  array  B[0 . . .  N],  where  N  is  the  time- 
horizon  of  the  simulation;  the  elements  of  the 
array  are  pairs  (r,  d)  where  r  is  a  system  state, 
and  d  is  a  set  of  events.  The  meaning  of  a 
behavior  is  as  follows.  If  B\j]  is  (r,d)  then 
the  state  of  the  system  at  time  j  is  r,  and  the 
set  of  events  that  occur  at  time  j  is  d.  At 
time  0,  each  process  is  in  its  initial  state  and 
initiates  its  initial  event.  Since  the  PPs  are 
deterministic,  the  system  admits  only  one  be¬ 
havior.  We  leave  it  to  the  reader  to  develop  an 
algorithm  to  compute  the  system  behavior  B 
given  the  specifications  of  the  PPs;  the  obvi¬ 
ous  solution  is  to  compute  B\j  + 1]  after  com¬ 
puting  jB[0  ...j].  This  solution  corresponds  to 
a  time-driven  simulation. 

Our  problem  is  to  design  a  simulator  to  de¬ 
termine  the  system  behavior  rapidly.  There 
are  no  restrictions  on  our  design.  For  instance 
we  permit  our  programs  to  fill  in  several  ele¬ 
ments  of  array  B  in  parallel,  or  to  start  with 
filling  in  the  middle  element. 

A  simulation  is  an  execution  of  the  simu¬ 
lator.  A  simulation  is  a  sequence  of  computa¬ 
tional  steps  and  we  do  not  interpret  the  mean¬ 
ing  of  a  computational  step.  A  simulation  may 
be  the  sequence  of  computational  steps  carried 
out  on  a  sequential  machine,  or  an  interleaving 
of  computational  tional  steps  carried  out  on  a 
distributed  machine  or  the  sequence  of  syn¬ 
chronous  operations  carried  out  by  a  parallel 
synchronous  machine.  A  point  in  the  simula¬ 
tion  is  an  integer  j,  which  represents  an  instant 
in  the  unfolding  of  the  simulation  in  which  the 
first  j  steps  of  the  simulation  have  been  exe¬ 
cuted  and  the  remaining  steps  have  not  been 
executed. 

A  simulator  consists  of  one  or  more  pro¬ 
cesses,  called  simulator  processes  (or  SPs)  to 
distinguish  them  from  physical  processes  (or 
PPs)  in  the  system  that  is  being  simulated. 
There  need  be  no  correspondence  between  SPs 


and  PPs.  In  a  sequential  simulation  there  may 
be  only  one  SP.  In  a  two  processor  machine 
there  may  be  two  SPs.  We  place  no  restric¬ 
tions  on  SPs  because  our  goal  is  to  obtain  a 
general  understanding  of  all  simulators,  rather 
than  to  understand  a  particular  implementa¬ 
tion  of  a  distributed  simulator.  (Note:  in  most 
implementations  there  is  one  SP  correspond¬ 
ing  to  each  PP,  but  there  is  no  reason  to  re¬ 
strict  our  design  to  have  a  correspondence  be¬ 
tween  processes  in  the  system  that  is  being 
simulated  and  processes  in  the  simulator.) 

Let  b  be  a  predicate  on  the  system  behavior. 
For  example,  b  may  be:  P[5]  is  (r,d)  where  d 
includes  the  event  ”firel” ,  and  the  state  of  PP 
0  in  r  is  “on”. 

Define  a  relation  “knows”  between  SPs, 
predicates  on  system  behavior  and  points  in 
the  computation,  as  follows: 

q  knows  b  at  j,  where  q  is  an  SP,  b  is  a 
predicate  on  the  behavior,  and  j  is  a  point  in 
the  simulation,  means  that  6  can  be  deduced 
given  only  the  state  of  q  at  j,  and  the  specifi¬ 
cation  for  q. 

EXAMPLE:  Consider  the  example  of  the 
time-driven  simulation  of  the  submarine  bat¬ 
tle.  Assume  that  there  is  one  SP  for  each  sub. 
An  SP  has  the  specification  of  one  sub,  but 
has  no  information  about  the  other.  If  the 
clock  has  value  t,  and  an  SP  has  obtained  the 
event  that  is  executed  by  the  other  sub  at  time 
t,  then  by  employing  its  function  /  its  state  at 
time  t  +  1  can  be  determined,  and  by  employ¬ 
ing  its  function  g  the  event  it  initiates  at  time 
t  +  1  can  be  determined.  Therefore  the  SP 
knows  its  state  at  time  t  +  1  and  it  knows  the 
event  it  initiates  at  time  t  +  1.  Unfortunately, 
the  SP  does  not  know  anything  about  the  be¬ 
havior  at  times  after  t +1  because  that  depends 
on  the  specification  of  the  other  sub.  To  gain 
knowledge  about  times  later  than  t  + 1  the  SP 
must  receive  information  from  the  other  SP. 


Therefore,  the  concept  of  knowledge  gives  us 
an  idea  of  the  amount  of  information  transfer 
required  in  this  case.  Informally  speaking,  in 
this  example  a  great  deal  of  information  trans¬ 
fer  is  required  because  a  process  has  limited 
knowledge. 

EXAMPLE:  Next  consider  the  job-shop 
example.  Assume  that  there  is  one  SP  for 
each  workstation  and  each  SP  has  the  speci¬ 
fication  of  the  corresponding  workstation,  but 
does  not  have  the  specification  of  any  other 
workstation.  Assume  that  at  some  point  in 
the  simulation  an  SP  simulating  a  workstation 
has  the  following  information:  a  job  requiring 
10  units  of  service  arrives  at  the  empty  work¬ 
station  at  time  5,  and  a  job  requiring  20  units 
of  service  arrives  at  the  workstation  at  time  6. 
At  this  point  in  the  simulation  the  SP  knows 
that  the  workstation  outputs  no  jobs  in  the  in¬ 
terval  (5,15),  it  outputs  one  job  at  15  and  the 
following  one  at  35.  Thus  the  SP  can  carry 
out  this  computation  without  receiving  infor¬ 
mation  from  other  SPs. 

EXAMPLE:  Consider  the  example  of  the 
battle  again,  except  that  one  SP  has  the 
specification  for  both  subs.  In  this  case,  the 
SP  knows  the  entire  system  behavior  ini¬ 
tially.  The  SP  needs  no  additional  informa¬ 
tion.  Thus  we  draw  a  distinction  between 
what  an  SP  knows  and  what  it  has  computed. 

In  designing  efficient  parallel  simulators,  we 
often  tradeoff  what  an  SP  knows  with  the 
amount  of  computation  it  does.  In  a  sequen¬ 
tial  simulation,  the  single  SP  knows  the  en¬ 
tire  behavior  initially,  and  an  execution  of  the 
simulator  consists  of  the  SP  computing  what 
it  knows.  In  a  time-driven  simulation,  with 
one  SP  per  PP,  an  SP  only  knows  the  be¬ 
havior  at  the  next  value  of  the  clock.  Again 
speaking  very  informally,  in  an  efficent  sim¬ 
ulation,  by  the  time  an  SP  computes  what 


it  knows,  additional  information  arrives  to  in¬ 
crease  its  knowledge.  If  an  SP’s  knowledge 
stays  ahead  of  what  it  has  computed  then  the 
SP  need  not  become  idle  waiting  for  informa¬ 
tion.  Some  of  the  tuning  that  is  done  in  mak¬ 
ing  distributed  systems  more  efficient — for  in¬ 
stance  in  “packaging”  the  simulation  of  several 
PPs  on  the  same  processor — consists  of  spec¬ 
ifying  SPs  so  that  an  SP’ a  knowledge  stays 
ahead  of  its  computation.  This  form  of  pack¬ 
aging  SPs  has  a  substantial  impact  on  the 
performance  of  a  distributed  simulator. 

Next  we  shall  extend  our  definition  of  knowl¬ 
edge  to  “conditional  knowledge”  and  then  at¬ 
tempt  to  use  our  insight  to  design  distributed 
simulators. 

4  CONDITIONAL  KNOWLEDGE 

Consider  the  example  of  the  submarine  bat¬ 
tle  once  again.  Suppose  the  specification  of 
subO  is  that  it  will  fire  its  first  missile  at  time 
10  if  it  is  not  destroyed  earlier.  Suppose  there 
is  an  SP  that  has  the  specification  of  subO,  and 
has  no  other  information.  What  does  the  SP 
know  when  the  simulation  is  initiated?  Ac¬ 
cording  to  our  definition  of  “knows” ,  when  the 
simulation  is  initiated,  the  SP  knows  the  state 
of  subO  at  time  0  and  the  event  initiated  by 
subO  at  time  0.  The  SP  does  not  know  the 
state  of  subO  at  time  1  because  it  does  not 
know  the  event  initiated  by  the  other  sub  at 
time  0.  In  particular  the  SP  does  not  know 
whether  subO  fires  it  first  missile  at  time  10. 
What  else  does  the  SP  know  initially?  The 
SP  knows 

“subO  is  not  destroyed  in  the  interval  (0,10) 
implies 

subO  fires  its  first  missile  at  time  10” . 

We  define  a  ternary  relation  “conditionally- 
knows”  between  an  SP,  two  predicates  on  the 


system  behavior  and  a  point  in  the  simulation: 

q  conditionally-knows  b  given  b'  at  j 
where  q  is  an  SP,  b  and  V  are  predicates  on 
the  system  behavior,  and  j  is  a  point  in  the 
simulation,  means 

q  knows  (6'  =►  6)  at  j.  In  our  sub¬ 
marine  example,  the  SP  conditionally-knows 
“subO  fires  its  first  missile  at  time  10”  given 
“subO  is  not  destroyed  in  the  interval  (0,  10)” 
at  0,  i.e.,  at  initiation  of  the  simulator.  The 
concept  of  conditional  knowledge  is  not  a  new 
concept  at  all;  but  we  dignify  knowledge  of 
predicates  of  the  form  implies  b”  with  a 
special  name,  because  this  form  of  knowledge 
plays  an  important  role  in  simulation. 

Conditional  knowledge  may  not  appear  use¬ 
ful  in  itself.  However,  as  a  simulation  pro¬ 
ceeds,  conditional  knowledge  can,  sometimes, 
be  converted  to  knowledge.  How  can  condi¬ 
tional  knowledge  be  converted  to  knowledge? 
Obviously,  if  q  conditionally-knows  b  given  V 
at  j,  and  q  knows  b'  at  j,  then  q  knows  b  at 
j.  Informally  speaking,  conditonal  knowledge 
and  some  knowledge  can  yield  more  knowl¬ 
edge.  Also  conditional  knowledge  and  more 
conditional  knowledge  can  yield  knowledge. 
For  example,  if  an  SP  conditionally-knows 
that  subO  fires  its  first  missile  at  time  10  given 
that  it  is  not  destroyed  earlier,  and  the  SP 
also  conditionally-  knows  that  subl  fires  its 
first  missile  at  time  20  given  that  it  is  not  de¬ 
stroyed  earlier,  then  the  SP  also  knows  that 
subO  fires  its  first  missile  at  time  10  (assum¬ 
ing  that  a  sub  is  destroyed  only  if  the  other 
fires).  Indeed,  this  conversion  of  conditional 
knowledge  to  knowledge  forms  the  basis  for 
sequential  simulation. 

Now  let  us  use  our  insight  to  explore  some 
methods  of  distributed  simulation. 


5  EXPLORING  METHODS  FOR  DIS¬ 
TRIBUTED  SIMULATION 


Our  goal  here  is  to  suggest  that  the  concepts 
of  knowledge  and  conditional  knowledge  may 
be  useful  in  the  design  of  distributed  simula¬ 
tions.  We  do  not  imply  that  the  algorithms 
proposed  here  are  better  than  others  or  even 
that  they  axe  original.  Another  goal  here,  is 
to  use  the  concept  of  knowledge,  coupled  with 
that  of  computation,  to  get  a  VERY  ROUGH 
idea  of  the  efficiency  of  an  approach  without 
doing  a  great  deal  of  empirical  testing. 

5.1  Each  SP  starts  with  a  different 
initial  state 

Consider  a  simulator  in  which  there  are 
two  SPs  where  SP0  has  the  specification 
for  all  PPs,  and  SP1  has  the  specifica¬ 
tion  for  all  PPs,  except  that  in  place  of 
the  initial  states  and  events,  SP1  has  some 
other  values — let’s  call  these  values  X.  SPl 
carries  out  a  simulation  starting  with  ini¬ 
tial  values  X.  Suppose  SPl  has  computed 
a  sequence  of  (states,  event-sets)  tuples — 
(rO,  dO),  (rl,  dl),  (r2,  d2) — where  (r0,d0)  cor¬ 
responds  to  X.  Now  SPl  conditionally-knows 
B\j  +  1  ..j  +  2]  =  (rl,dl),(r2,  d2)  given  that 
B[j]  =  (rO,  dO),  where  B  is  the  system  be¬ 
havior.  In  other  words,  SPl  conditionally- 
knows  the  behavior  at  j  +  1  and  j  +  2  given 
that  the  behavior  at  j  is  the  initial  value  em¬ 
ployed  by  SPl.  Here  SP0  knows  the  entire 
behavior — it  needs  no  information  from  SPl. 
The  following  approach  to  distributed  simu¬ 
lation  can  be  taken.  Both  SB’s  compute  in 
parallel.  If  SP0  has  computed  a  j  such  that 
B\j\  =  (rO,  dO),  then  it  sends  the  value  of  j  to 
SPl.  Upon  receiving  this  value,  conditional 
knowledge  in  SPl  is  converted  to  knowledge, 
and  this  knowledge  has  already  been  com- 


puted  by  SP 1. 

This  example  illustrates  that  it  may  be 
efficient  for  an  SP  to  compute  conditional- 
knowledge  because  the  values  computed  may 
turn  out  to  be  knowledge. 

5.2  Messages  with  conditional  and 
unconditional  information 

Consider  a  closed  loop  of  two  workstations. 
The  output  of  each  station  feeds  the  input  of 
the  other.  Each  station  has  a  single  server. 
There  are  a  fixed  number  of  jobs  in  the  sys¬ 
tem  and  each  job  cycles  repeatedly  through 
the  two  work-  stations.  There  are  two  priori¬ 
ties  of  jobs.  A  job  may  switch  priority  when  it 
completes  service  at  a  station.  Whether  a  job 
switches  priority  is  determined  by  the  work¬ 
station  at  which  it  completes  service. 

The  simulator  has  two  SPa,  one  for  each 
workstation.  Suppose  an  SP  knows  that  at 
time  t  a  workstation  is  serving  a  high-priority 
job  that  requires  t'  additional  units  of  service; 
then  the  SP  knows  that  the  workstation  will 
output  the  job  at  time  t  + 1' . 

Now  suppose  that  an  SP  knows  that  at  time 
t  a  workstation  is  serving  a  low-priority  job 
that  requires  t'  additional  units  of  service,  and 
no  high-priority  job  arrives  at  the  station  at 
time  t.  Then,  if  t'  >  1  and  all  jobs  require  at 
least  one  unit  of  service,  then  the  SP  knows 
that  the  work-  station  does  not  output  a  job 
at  time  t  +  1;  this  is  because  it  cannot  out¬ 
put  the  low-priority  job  that  it  is  serving  at 
least  until  t  + 11,  and  if  a  high-priority  job  ar¬ 
rives  at  time  t  +  1,  it  cannot  output  that  for 
at  least  one  additional  time  unit.  The  SP  also 
conditionally-knows  that  the  next  job  out-  put 
by  the  workstation  is  at  time  t  + 1'  given  that 
no  high-  priority  job  arrives  at  the  workstation 
in  the  interval  (t,t  + 1'). 


Let  us  design  the  simulator  so  that  the  SPa 
exchange  information  about  both  knowledge 
and  conditonal  knowledge.  For  example  as¬ 
sume  that  SPQ,  SPl  are  simulating  worksta¬ 
tions  WO,  Wl  respectively.  Assume  that  SPl 
conditionally-knows  that  Wl  outputs  no  high- 
priority  job  in  the  interval  (t,?)  given  that 
WO  outputs  no  high-priority  job  in  that  inter¬ 
val.  Also  assume  that  SPQ  sends  a  message  to 
SPl  after  receiving  which,  SPl  conditionally- 
knows  WO  outputs  no  job  in  (i+1,  t+t')  given 
Wl  outputs  no  high-priority  job  in  that  inter¬ 
val.  Upon  receipt  of  the  message,  SPl  knows 
that  Wl  outputs  no  high-priority  job  in  the 
interval 

In  this  example,  the  exchange  of  both  condi¬ 
tional  and  unconditional  information  is  helpful 
in  allowing  the  simulation  to  progress. 

This  approach  can  be  extended  to  an  ar¬ 
bitrary  network.  In  the  worst  case,  this  ap¬ 
proach  reduces  to  a  sequential  simulation. 
Therefore,  this  approach  has  the  advantage 
of  significant  speed-up  on  parallel  machines  in 
some  applications  and  of  being  no  worse  than 
sequential  simulation  for  all  applications. 


5.3  Deadlock 


It  is  interesting  to  interpret  deadlock,  from 
the  point  of  view  of  knowledge.  A  pro¬ 
cess  waits  when  it  has  computed  everything 
it  knows.  All  processes  wait — i.e.,  remain 
deadlocked — if  each  process  has  computed  ev¬ 
erything  it  knows.  “Breaking  deadlock”  means 
that  some  external  process  supplies  informa¬ 
tion  to  the  set  of  deadlocked  processes  that 
results  in  at  least  one  of  them  gaining  knowl¬ 
edge,  and  the  process  then  continues  compu¬ 
tation.  The  [Chandy,  Misra  1981]  paper  cam 
be  interpreted  in  this  way. 


5.4  Optimistic  Computations 


Many  ways  of  implementing  optimistic  com¬ 
putations  [Jefferson  1985]  become  apparent 
when  we  try  to  understand  distributed  sim¬ 
ulation  from  the  point  of  view  of  conditional 
knowledge. 

A  process  that  has  computed  everything  it 
knows  may  wait  to  receive  additional  infor¬ 
mation,  or  it  may  proceed  to  compute  con¬ 
ditional  knowledge.  A  pessimistic  computa¬ 
tion  is  one  where  a  process  only  computes 
what  it  knows.  An  optimistic  computation 
is  one  in  which  a  process  may  also  com¬ 
pute  what  it  conditionally-knows.  (Pessimistic 
computations  may  also  use  conditional  knowl¬ 
edge,  because — as  described  in  Section  5.2 — 
conditional  knowledge  may  be  converted  to 
knowledge.)  The  advantage  of  optimistic  com¬ 
putations  is  that  an  SP  never  waits. 

Consider  a  simulator  in  which  all  condi¬ 
tional  knowledge  that  is  computed  is  clearly 
identified  as  conditonal  knowledge  (as  dis¬ 
tinguished  from  knowledge).  For  example  if 
a  simulator  of  a  job-shop  determines  that  a 
work-  station  W  outputs  a  job  at  time  t  given 
that  it  receives  no  high-priority  job  in  the  in¬ 
terval  (a? ,<'),  then  this  result  is  stored  in  mem¬ 
ory  exactly  in  the  form  of  conditional  knowl¬ 
edge,  i.e.,  in  the  form  b  given  V  where  b  is  “W 
out-  puts  a  job  at  time  t”  and  V  is  “W  receives 
no  high-  priority  job  in  the  interval  (x,t)”. 
Now  suppose  that  as  the  simulation  unfolds,  it 
is  determined  that  W  receives  a  high-priority 
job  in  the  interval  (x,t).  Does  the  conditioned 
knowledge  have  to  be  discarded? 

There  is  no  reason  to  discard  conditional 
knowledge  except  the  practical  reason  that 
there  is  insufficient  memory  to  store  the  con¬ 
ditional  knowledge  that  has  been  computed. 
It  is  not  as  though  conditional  knowledge  be¬ 


comes  “false”  and  therefore  must  be  erased  to 
restore  the  system  to  a  correct  state.  In  our 
example,  it  is  always  the  case  that  W  outputs 
a  job  at  time  t  given  that  it  receives  no  high- 
priority  job  in  the  interval  (x,t),  even  if  W 
does  receive  a  high-priority  job  in  the  interval. 

What  are  the  disadvantages  of  optimistic 
computations?  In  many  simulators,  an  SP 
does  not  record  everything  it  knows.  For  in¬ 
stance,  an  SP  may  not  recored  the  entire  state 
of  a  workstation  because  it  may  not  be  neces¬ 
sary  to  determine  the  entire  behavior — it  may 
be  sufficient  to  determine  some  attributes  of 
the  behavior,  such  as  “what  is  the  average 
time  spent  by  jobs  in  the  job-shop”?  When  an 
SP  starts  computing  conditional  knowledge,  it 
must  distinguish  conditional  knowledge  from 
knowledge;  in  particular,  it  may  have  to  save 
the  entire  state  of  the  processes  that  it  is  sim¬ 
ulating,  because  this  state  is  knowledge  rather 
than  conditional  knowledge.  Saving  the  state 
(and  then  copying  the  saved  state)  may  be  a 
considerable  overhead.  The  extent  of  the  over¬ 
head  depends  on  the  system  being  simulated. 
The  overhead  may  outweigh  the  advantages  of 
not  waiting. 

5.5  Processing  Conditional  Knowl¬ 
edge 

Conditional  knowledge  can  be  processed  in 
many  ways.  Consider  the  following  example. 
The  system  to  be  simulated  consists  of  three 
processes  u.i,  i  =  0, 1, 2,  connected  in  a  cy¬ 
cle,  where  the  events  initiated  by  process  u.i 
may  modify  the  states  of  processes  u.i  and 
u.(i  +  l)mod3.  There  are  three  SPs:  SP.i, 
i  =  0, 1,2,  corresponding  to  the  u.i.  Suppose 
SP.I  receives  a  message  from  SP. 0  that  the 
first  event  u.O  initiates  is  at  time  10  given  that 
u.2  initiates  no  event  before  time  8.  Suppose 


that  initially  SP.l  conditionally-knows  u.l  ini¬ 
tiates  its  first  event  at  time  20  given  u.O  ini¬ 
tiates  no  event  before  time  5.  Upon  receipt 
of  the  message  from  5 P.0,  SP.l  conditionally- 
knows  u.l  initiates  its  first  event  at  time  20 
given  u.2  initiates  no  event  before  time  5  (be¬ 
cause  if  u.2  initiates  no  event  before  time  5 
then — from  the  message  from  SP. 0 — u.O  ini¬ 
tiates  no  event  before  time  5  in  which  case — 
from  SP.l' a  initial  conditional  knowledge — u.l 
initates  its  first  message  at  time  20).  SP.l 
sends  this  conditional  knowledge  to  SP. 2. 

Now  suppose  that  initially  SP.  2 
conditionally-knows  that  u.2  initiates  its  first 
event  at  time  12  given  u.l  initiates  no  event  be¬ 
fore  time  9.  Upon  receiving  the  message  from 
SP.l  that  u.l  initiates  its  first  event  at  time 
20  given  u.2  initiates  no  event  before  time  5, 
SP. 2  gains  the  knowledge  that  u.2  initiates  its 
first  event  at  time  12. 

This  little  example  illustrates  that  there  are 
many  ways  of  processing  conditional  knowl¬ 
edge. 

6  CONCLUSION 

The  goal  of  this  paper  is  to  suggest  that 
we  may  have  been  unnecessarily  restrictive  in 
the  way  we  think  about  distributed  simula¬ 
tion.  We  attempted  to  develop  a  framework  in 
terms  of  knowledge  to  free  ourselves  from  un¬ 
necessarily  restrictive  modes  of  thought.  As  a 
consequence,  different  methods  of  distributed 
simulation  suggested  themselves.  Of  course, 
the  methods  could  have  been  thought  of  with¬ 
out  the  aid  of  the  concept  of  knowledge.  We 
hope,  however,  that  the  concept  is  useful  in 
designing  simulators. 

In  our  opinion  the  success  of  distributed 
simulation  depends,  in  large  part,  on  our  abil¬ 
ity  to  exploit  properties  of  the  system  that  is 


being  simulated.  For  example  using  only  con¬ 
ditional  knowledge  (as  in  sequential  simular- 
tion),  or  using  only  unconditional  knowledge 
(as  in  some  methods  of  distributed  simula¬ 
tion),  does  not  exploit  properties  of  the  system 
to  the  fullest.  We  think  that  there  is  a  better 
chance  of  achieving  efficiencies  on  parallel  pro¬ 
cessors  by  employing  both  forms  of  knowledge. 
Indeed,  there  may  be  other  types  of  knowledge 
that  can  be  exploited  for  specific  applications. 

A  problem  that  has  been  receiving  a  lot  of 
attention  lately  is  that  of  designing  “general 
purpose”  distributed  discrete-event  simulators 
in  which  the  processes  in  the  system  to  be  sim¬ 
ulated  are  treated  as  “black  boxes”.  In  our 
opinion,  the  efficiency  of  such  simulators  de¬ 
pends  on  the  interfaces  to  the  black  boxes. 
If  a  black  box  merely  specifies  what  the  cor¬ 
responding  process  does  during  the  next  in¬ 
cremental  interval  of  time,  then  the  simula¬ 
tion  reduces  to  a  time-driven  simulation.  If  a 
black  box  only  specifies  unconditional  knowl¬ 
edge  then  the  resulting  simulator  may  be  inef¬ 
ficient,  as  suggested  in  our  battlefield  example. 
If  a  black  box  only  specifies  conditional  knowl¬ 
edge,  then  the  resulting  simulator  may  be  inef¬ 
ficient,  as  suggested  by  our  job-shop  example. 

The  problem  is  to  trade-off  the  “general  pur¬ 
pose”  nature  of  the  simulator  on  the  one  hand 
with  efficiency.  If  we  insist  that  a  general- 
purpose  simulator  is  one  that  has  very  little  in¬ 
formation  about  the  system  being  simulated, 
then  we  may  have  to  pay  the  price  in  terms 
of  efficiency.  On  the  other  hand,  we  cannot 
afford  to  propose  a  new  method  for  simula¬ 
tion  for  each  new  application.  The  concepts 
of  knowledge  and  conditional  knowledge  may 
help  in  defining  an  interface  that  does  not  ex¬ 
pose  too  much  of  what  is  in  the  black  boxes, 
and  also  results  in  simulators  that  are  efficient. 


This  paper  is  extremely  informal.  A  great 
deal  of  work  remains  to  be  done  to  define  the 
concepts  proposed  here  formally,  to  explore 
new  concepts  that  will  help  further  in  freeing 
us  from  narrow  modes  of  thought  about  simu¬ 
lation,  in  specifying  the  interfaces  to  the  black 
boxes  of  general-  purpose  simulators,  and  in 
implementing  the  ideas  on  parallel  machines. 
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